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Executive Summary 

This evaluation study provides information about the implementation and outcomes 
of the 21st Century Community Learning Centers (21st CCLC) program in West Virginia, from 
September 2014 through May 2015. 

Method of study. The report draws on information from online surveys of 23 directors 
of 21st CCLC programs and from school teachers for 929 of the 11,299 participating students. 
It also draws on West Virginia General Summative Assessment (WVGSA) scores provided by 
the West Virginia Department of Education (WVDE) for students who participated in a 21st 
CCLC program for at least 30 days (1,864), and a matching group of 1,864 nonparticipants 
used as a control. The students were grouped by grade level and compared in mathematics 
and English/language arts (ELA) using scale score means for Grades 4-11. 

Findings. Most participating students were in the elementary grades. The mean num- 
ber of days students attended ranged from about 2 to 108 days, depending on the program. 
Teachers perceived the greatest improvements in participating students’ behaviors related to 
turning in homework on time, participating in class, completing homework to teachers’ satis- 
faction, and academic performance. Regarding 21st CCLC program volunteers, the largest 
sources were K-12 service learning programs (36.8%), parents (20.2%), and higher education 
service learning programs (14.8%). The groups with which program directors reported the 
greatest level of success were higher education service learning programs (3.8 on a 4.0 scale), 
community organizations (3.6 of 4.0), and K-12 service learning programs (3.6 of 4.0). Re- 
garding work with partners, the most frequent types of support received were program re- 
sources (28.2%), programming (22.1%), and joint planning (18.0%). 

With the exception of a few activities, all program directors who had engaged in the 
various partnership activities considered them to be effective. Program directors said they 
needed more professional development in program sustainability, more technical assistance 
in program evaluation, and more information resources in STEM/STEAM. As for parent and 
community involvement, more than 60% of program directors indicated they either had no 
family component or they had only slight success in their efforts to involve parents/guardians 
or other adult community members. In responses to open-ended questions, program directors 
indicated programs for students and better student attendance/participation were the most 
successful. The program directors felt the greatest challenges were personnel issues, parent 
engagement and support, and funding and sustainability. Lastly, program directors were 
asked to make recommendations for how to improve the program for the future. Most pro- 
gram directors who commented wrote about redundancies in reporting, the hard-to-use 
WVEIS 21st CCLC data collection interface, and the inability to utilize data entered. 

The quasi-experimental study in mathematics using scale score means for Grades 4- 
11 showed the observed differences were not statistically significant. The quasi-experimental 
study in ELA using scale score means for Grades 4-11 also showed that the observed differ- 
ences were not statistically significant. 
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Limitations of study. We cannot assume that the 21st CCLC attendance was a key fac- 
tor in the improvement of behaviors perceived by teachers. Some results are based on percep- 
tions of teachers and program directors. 
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Introduction 


The West Virginia Department of Education (WVDE) has implemented a program, the 
21st Century Community Learning Centers (21st CCLC), to provide opportunities for commu- 
nities to establish or expand activities in communities that 

1 . provide opportunities for academic enrichment, including providing tutorial services 
to help students— particularly students who attend low-performing schools— to meet 
state and local student academic achievement standards in core academic subjects, 
such as reading and mathematics; 

2. offer students a broad array of additional services, programs, and activities, such as 
youth development activities, drug and violence prevention programs, counseling pro- 
grams, art, music, and recreation programs, technology education programs, and 
character education programs, that are designed to reinforce and complement the reg- 
ular academic program of participating students; and 

3 . offer families of students served by community learning centers opportunities for 
literacy and related educational development. 

The 21st CCLC program was authorized under Title IV, Part B, of the Elementary and 
Secondary Education Act, as amended by the No Child Left Behind Act of 2001, which trans- 
ferred administration of the program from the U. S. Department of Education to state educa- 
tion agencies. 

WVDE makes competitive local grants based on available federal funding to eligible 
organizations to support the implementation of community learning centers that will aid stu- 
dent learning and development. Eligible applicants are public and private agencies, city and 
county governmental agencies, faith-based organizations, institutions of higher education, 
and for-profit corporations. 

The purpose of this evaluation study is to provide information about the implementa- 
tion and outcomes of the 21st CCLC program in West Virginia, during the period from Sep- 
tember 2014 through May 2015. 


Evaluation Questions 

This evaluation study addresses several broad evaluation questions: 

EQi Student participation and impacts. Which students were referred to 21st CCLC, for 
what reasons, at what levels of participation, and to what effect? 

EQ2 Volunteers and partnerships. How did programs operate with regard to volunteers, 
partnerships, and information sharing? 

EQ3 Professional development and technical assistance. How well did professional de- 
velopment and technical assistance support 21st CCLC programs, which formats are 
preferred, and what topics are most needed? 
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EQ4 Parent and community involvement. What was the level of success in involving 
parents and community members? 

EQ5 Improvement and accountability processes. How helpful to 21st CCLC programs 
were improvement and accountability processes? 

EQ6 Successes, challenges, and recommendations. What do program directors view as 
their major successes, challenges, and recommendations for the future of the pro- 
gram? 

Methods 

We addressed EQi using responses to the 21st Century Community Learning Centers 
(21st CCLC) Teacher Survey from teachers who taught participating students during the reg- 
ular school year; their responses were analyzed using descriptive statistics. We determined 
levels of participation in each program based on program directors’ reports of student attend- 
ance in the 21st CCLC database; these reports are also used as the source of data about the 
number of days of participation (dose strength) for individual students. Using quasi-experi- 
mental methods we tested impacts on West Virginia General Summative Assessment 
(WVGSA) scores provided by the West Virginia Department of Education (WVDE) for stu- 
dents who participated in a 21st CCLC program for at least 30 days. Data were collected from 
the 21 st CCLC and West Virginia Education Information System (WVEIS) reporting systems. 

The remaining five evaluation questions (EQ2-EQ6) were addressed using responses 
from the 21st CCLC Program Directors Survey, which were analyzed using descriptive statis- 
tics. This survey questionnaire was streamlined in April 2015 to reduce the burden on program 
directors while still collecting sufficient data to adequately address each of the evaluation 
questions. 

A summary of the methods and data sources used in this study can be found in Table 
1. A more detailed description of the methods is included in Appendix A. Survey instruments 
are in Appendix B, and informed consent forms are in Appendix C. 

Study Participants and Sampling 

The study included student subjects, whose regular school-day teachers were con- 
tacted for their observations about changes in their behavior and performance (see Teacher 
Survey in Appendix B). The study used students’ WVGSA scores in a quasi-experimental anal- 
ysis. In this analysis, we selected the student treatment group contingent upon their having 
participated in a 21st CCLC for at least 30 days; we also used scores from a matching control 
group of students not known to have been participants. The only other participants and sub- 
jects in the study were 21st CCLC program directors, all of whom were contacted to participate 
in the study. 


2 / 21st Century Community Learning Centers, 2014-2015 


Methods 


Table 1. Summary of Evaluation Questions and Data Collection Methods 

Method of analysis/ 

Evaluation question data source Results reported 


EQ1. Student participa- 
tion and impacts. 
Which students were 
referred to 21st CCLC, 
for what reasons, at 
what levels of partici- 
pation, and to what ef- 
fect? 


Descriptive statistics/ Among students who had participated in a 21st CCLC for 
Online 21 st CCLC at least 30 da Y s ' 

Teacher Survey • Which behaviors did teachers identify as needing im- 

provement? 

• Which behaviors did teachers report as having im- 
proved? 

Descriptive statistics/ • What was the distribution of students by grade level? 

WVDE 21 st CCLC • Across programs, what was the level of participation 

database (dose strength)? 


Quasi-experimental 

study/ 

General Summative 
Assessment scores and 
WVDE 21 st CCLC 
database 


EQ2. Volunteers and 
partnerships. How did 
programs operate with 
regard to volunteers 
and partnerships? 


Descriptive statistics/ 
Online 21 st CCLC 
Program Director 
Survey 


Among students who had participated in a 21st CCLC for 

at least 30 days, 

• What was the impact of 21st CCLC participation on 2- 
year English/language arts (ELA) and mathematics 
gains?* 

• What was the impact of 21st CCLC participation on 
end-of-year ELA and mathematics achievement?* 

• What were the year-to-year changes in ELA and 
mathematics achievement for each group ( 21st CCLC 
participants and nonparticipants) independently?* 

• Were ELA and mathematics gains experienced by 
21st CCLC participants significantly different from 
those gains experienced by nonparticipants?* 

• How many volunteers were involved in programs and 
from which sources? 

• At what level of success did program directors work 
with each source of volunteers? 

• How many partners did programs work with, and 
what was the nature of partners' support? 

• How effective were collaborations with partners? 


EQ3. Professional de- Descriptive statistics/ 

velopment and tech- 0n / ine 2 v* CCLC 

nical assistance. How Program Director 

well did professional Survey 

development and tech- 
nical assistance support 
21st CCLCs, which for- 
mats are preferred, and 
what topics are most 
needed? 


• What was the quality of professional development 
offered by counties and RESAs on various topics; and 
what is the ongoing need for more on these topics 
from these sources? 

• What was the quality of professional development 
offered by WVDE on various topics; and what is the 
ongoing need for more on these topics from this 
source? 

• What was the quality of professional development 
offered by the U.S. Department of Education on vari- 
ous topics; and what is the ongoing need for more on 
these topics from this source? 

• How helpful were various forms of technical assis- 
tance? Continued next page 
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Table 1. Summary of Evaluation Questions and Data Collection Methods 

Evaluation question 

Method of analysis/ 

data source 

Results reported 

EQ4. Parent and com- 
munity involvement. 
What was the level of 
success in involving 
parents and community 
members? 

Descriptive statistics/ 

Online 21 st CCLC 
Program Director 
Survey 

• How successful were programs in involving parents, 
guardians, and community members? 

• How many adults were involved in 21st CCLC activi- 
ties and what was the nature of their involvement? 

EQ5. Improvement and 
accountability pro- 
cesses. How helpful to 
21st CCLCs were im- 
provement and ac- 
countability processes? 

Descriptive statistics/ 

Online 21 st CCLC 
Program Director 
Survey 

• How helpful was the continuous improvement pro- 
cess for after school (CIPAS)? 

• How helpful were the WVDE monitoring visits? 

EQ6. Successes, chal- 
lenges, and recommen- 
dations. What do 
program directors view 
as their major suc- 
cesses, challenges, and 
recommendations for 
the future of the pro- 
gram? 

Descriptive statistics/ 

Online 21 st CCLC 
Program Director 
Survey 

• What did program directors view as their major suc- 
cesses? 

• What did program directors view as their major chal- 
lenges? 

• What specific professional development or technical 
assistance topics would you find most helpful for 
WVDE staff to deliver during the upcoming school 
year? 

• What recommendations did program directors have 
for improving the 21st CCLC program? 

*ln 2014-2015 this analysis will not be available due to the change from WESTEST 2 to the Smarter 
Balanced-based general summative assessment. These analyses will resume in 2015-2016 when two years 
of student scores will once again be available. 


Results 

Approximately 11,229 students were served by the West Virginia 21st Century Com- 
munity Learning Centers (21st CCLC) program. We received 929 teacher survey responses, 
which were used for all analyses in response to EQi except for determining grade level. 

We received responses from all 23 program directors who were contacted for the pro- 
gram director survey and all 23 program directors responded to the request for information 
about student attendance. 
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EQ1 Student Participation and Impacts 


Which students were referred to 
21st CCLC,for what reasons, at 
what levels of participation, and 
to what effect? 

Student participation by grade 
level 


Most student partici- 
pants— 6,742 of 11,299 or 60.7% 
were in elementary school (Grades 
PK-5). An additional 2,798 or 
24.8% were in middle school 
(Grades 6-8), and 1,757 or 15.6% 
were in high school (Grades 9-12) 
in 2014-2015. See Figure 1. 

Student behaviors in need of 
improvement 

Teachers rated students in 
terms of their need for improve- 
ment on 10 selected behaviors. 
Teachers were invited to select all 
behaviors relevant to each stu- 
dent. Figure 2 illustrates the per- 
centage of all 21st CCLC-enrolled 
students for whom we have re- 
ports indicating the need for im- 
provement in each behavior. The 
top five behaviors teachers indi- 
cated students needed improve- 
ment were (descending order) (a) 
completing homework to teach- 
ers’ satisfaction, (b) academic per- 
formance, (c) turning in home- 
work on time, (d) being attentive 
in class, and (e) coming to school 
motivated to learn. 
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Figure 1. Percentage of Student Participation by Grade Level 
Data source: 2015 CCLC teacher survey May-June 2015 
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Figure 2. Percentage of CCLC Students by Behaviors Needing 
Improvement 

Data source: 2015 CCLC teacher survey May-June 2015 
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Student change in behaviors 

Teachers also reported 
on how they viewed students’ 
change in key behaviors by the 
end of the year. 

Figure 3 displays the 
percentage of students teachers 
thought had improved, stayed 
the same, or worsened for each 
of the rated behaviors. Very few 
students were rated by their 
teachers as having declined in 
the level of their behavior. 

Levels of participation (dose 
strength) 

All 23 program direc- 
tors reported the total number 
of days individual student par- 
ticipants attended each 21st 
CCLC program (i.e., dose 
strength). Based on these data, 
we calculated the average num- 
ber of days attended per student for each program (see Table 2). Average attendance rates 
ranged from 1 to 108.4 days. 


Turning homework in on time 
Participating in class 
Completing homework to your... 
Academic performance 

/> 

o Behaving well in class 

> 

2 

5 Getting along well with other... 
n 

Coming to school motivated to learn 
Being attentive in class 
Attending class regularly 
Volunteering 


76 . 7 % 

14 . 4 % 


76 . 0 % 

17 . 0 % 


75 . 8 % 

16 . 4 % 


73 . 2 % 

17 . 8 % 



70 . 9 % 

16 . 0 % 



69 . 9 % 

18 . 7 % 



68 . 3 % 

22 . 6 % 


65 . 1 % 

22 . 6 % 


63 . 4 % 

27 . 7 % 


61 . 9 % 

33 . 5 % 


0 % 50 % 100 % 

Percent of students 

Showing improvement ■ No Improvement ■ Declining 


Figure 3. Percentage of Students That Teachers Assessed as 
Showing Behavior Improvement, No Improvement, or 
Decline. 

Data source: 2015 CCLC teacher survey May-June 2015 


Table 2. Program Attendance Dose Strength (Days per Student) 




Number 

Total 

Average 

Standard 



of 

number of 

number of 

deviation 

21st CCLC programs 

County(ies) 

students 

days 

days 

in days 

Total 


10,626 

353,488 

33.3 

38.7 

After School Explorers- 
Preston County 

Preston 

525 

23,788 

45.3 

27.1 

BLAST 

Braxton, Fayette, 

Nicholas, and Greenbrier 

247 

11,012 

44.6 

24.7 

Boys and girls club of the 
Eastern Panhandle Center 

Berkeley 

188 

6,797 

36.2 

26.9 

Charleston Extended 
Learning Centers-Bob 
Burdette Center 

Kanawha 

196 

21,241 

108.4 

58.1 

CONNECTIONS 

Kanawha 

42 

74 

1.8 

2.6 

DREAMS1 

McDowell 

211 

4,361 

20.7 

23.5 

DREAMS2 

McDowell 

878 

22,457 

25.6 

20.1 

FAST 

Fayette 

230 

3,642 

15.8 

15.0 


Table 2 continues on next page 
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Table 2. Program Attendance Dose Strength (Days per Student) 


21st CCLC programs 
HRDF-EEFRC The 

Challenge of Champions 
Afterschool Program 

County(ies) 

Kanawha 

Number 

of 

students 

133 

Total 

number of 
days 
8,104 

Average 
number of 
days 
60.9 

Standard 

deviation 
in days 
56.4 

Jackson County Patch 

Jackson 

404 

30,499 

75.5 

32.4 

Kaleidoscope Community 
Learning Centers 

Monongalia 

96 

117 

1.2 

0.5 

KidREACH World Vision 

Barbour 

135 

7,037 

52.1 

36.4 

Lincoln County 21st CCLC 

Lincoln 

358 

7,248 

20.2 

23.2 

Mason County Patch 

Mason 

439 

6,643 

15.1 

10.0 

Mingo County Youth 
Education Program 

Mingo 

77 

1,515 

19.7 

25.0 

Monongalia County 

Schools Kaleidoscope 

Monongalia 

35 

35 

1.0 

0.0 

Morgantown Connections 

Monongalia 

55 

5,373 

97.7 

47.9 

Mountaineer Boys & Girls 
Club 

Monongalia 

94 

2,831 

30.1 

29.0 

Partnership of African 

Kanawha 

158 

12,595 

79.7 

61.8 


American Churches' 


Communities Closing the 
Gap 


Partnership of African- 
American Churches-4 

sites 

Kanawha 

79 

242 

3.1 

1.5 

PATCH Ravenswood 

Jackson 

85 

2,302 

27.1 

12.9 

Project GOAL 

Boone 

199 

2,901 

14.6 

17.0 

Project ISAAC 

Barbour, Harrison, 

Marion, and Upshur 

418 

10,794 

25.8 

26.3 

Ritchie County STARS 

Ritchie 

181 

9,099 

50.3 

24.6 

Roane County PATCH 

Roane 

667 

12,956 

19.4 

22.4 

Salvation Army Boys & 
Girls Club St. Albans 

Kanawha 

98 

3,712 

37.9 

2.1 

SPLASH 

Braxton, Webster, 
Pocahontas, and Nicholas 

397 

4,448 

11.2 

12.4 

Spring Hill Elementary 

Kanawha 

376 

7,931 

21.1 

31.1 

STARS 

Ritchie 

217 

2,030 

9.4 

7.6 

SUCCESS 

Braxton, Fayette, 

Nicholas, and Webster 

586 

7,195 

12.3 

13.9 

Wayne County 

Community Learning 
Centers 

Wayne 

2,649 

106,693 

40.3 

48.1 

West Virginia Dreamers 
After School Program at 
South Park 

Kanawha 

80 

3,632 

45.4 

33.1 

West Virginia Dreamers 

Logan 

93 

4,184 

45.0 

26.1 


Afterschool at MAN (Man 
Elementary) 

Data source: 21st CCLC program director-supplied lists of students who participated in their program 
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Impact of student participation 

In 2015 a new online assessment, the West Virginia General Summative Assessment 
(WVGSA), was administered to students in Grades 3-11 for both mathematics and English/ 
language arts (ELA). Consequently only one year of assessment data were available for analy- 
sis and only one impact question could be addressed, that is, “Among students who had par- 
ticipated in a 21 st CCLC for at least 30 days (experimental group), were their ELA and 
mathematics gains significantly different from gains experienced by nonparticipants (control 
group)?” 

The control group students were selected using propensity score matching (PSM) on 
specific criteria (see Appendix A, page 19 for a detailed description of methods used). There 
were 1,864 student in both the experimental and control groups in Grades 3-11. 

Table 3 presents the results of independent samples t tests used to determine the sta- 
tistical significance of differences in mathematics mean scale scores between Group 1 (21st 
CCLC participants) and Group o (nonparticipants) for Grades 4-11. In no case were the slight 
differences observed statistically significant. 


Table 3. Comparison Between 21 st CCLC Participants and Nonparticipants in Mathematics 


Group 

Grade 

N 

Mean 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

l-21st CCLC participants 

4 

507 

2,452.2 

72.9 

.273 

1,012 

.785 

O-Nonparticipants 

4 

507 

2,450.9 

76.3 




l-21st CCLC participants 

5 

465 

2,472.1 

82.0 

-.302 

928 

.762 

O-Nonparticipants 

5 

465 

2,473.7 

79.1 




l-21st CCLC participants 

6 

350 

2,466.0 

95.4 

-.110 

698 

.912 

O-Nonparticipants 

6 

350 

2,466.8 

90.0 




l-21st CCLC participants 

7 

195 

2,482.4 

90.6 

-.423 

388 

.673 

O-Nonparticipants 

7 

195 

2,486.3 

89.6 




l-21st CCLC participants 

8 

176 

2,483.7 

97.8 

.394 

350 

.694 

O-Nonparticipants 

8 

176 

2,479.5 

101.6 




l-21st CCLC participants 

9 

84 

2,472.8 

83.5 

-.387 

166 

.699 

O-Nonparticipants 

9 

84 

2,478.2 

97.6 




l-21st CCLC participants 

10 

48 

2,492.2 

80.0 

.163 

94 

.871 

O-Nonparticipants 

10 

48 

2,489.2 

98.1 




l-21st CCLC participants 

11 

39 

2,512.4 

106.4 

.771 

76 

.443 

O-Nonparticipants 

11 

39 

2,494.7 

96.1 




*p<.05 for significance 


Table 4 presents the results of independent t tests used to determine the statistical 
significance of differences in ELA mean scale scores between Group 1 (21st CCLC participants) 
and Group o (nonparticipants) for Grades 4-11. In no case were the observed differences sta- 
tistically significant, although for Grade 4 the difference between the groups approached sta- 
tistical significance, with the nonparticipant group scoring higher than the 21st CCLC 
participants. 
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Results 


Table 4. Comparison Between 21 st CCLC Participants and Nonparticipants in English/Language Arts 


Group 

Grade 

N 

Mean 

Std. 

deviation 

t 

df 

Sig. (2 
tail)* 

l-21st CCLC participants 

4 

507 

2,444.8 

88.5 

-1.919 

1012 

.055 

O-Nonparticipants 

4 

507 

2,455.5 

88.5 




l-21st CCLC participants 

5 

465 

2,488.3 

85.5 

-.306 

928 

.760 

O-Nonparticipants 

5 

465 

2,490.0 

85.9 




l-21st CCLC participants 

6 

350 

2,491.8 

88.9 

.190 

698 

.849 

O-Nonparticipants 

6 

350 

2,490.5 

83.9 




l-21st CCLC participants 

7 

195 

2,512.8 

91.2 

-.644 

388 

.520 

O-Nonparticipants 

7 

195 

2,518.8 

90.4 




l-21st CCLC participants 

8 

176 

2,521.8 

82.5 

-.500 

350 

.617 

O-Nonparticipants 

8 

176 

2,526.3 

84.5 




l-21st CCLC participants 

9 

84 

2,513.5 

95.6 

-.973 

166 

.332 

O-Nonparticipants 

9 

84 

2,527.9 

96.2 




l-21st CCLC participants 

10 

48 

2,528.4 

99.3 

-.217 

94 

.829 

O-Nonparticipants 

10 

48 

2,532.7 

97.5 




l-21st CCLC participants 

11 

39 

2,557.5 

105.4 

1.772 

76 

.080 

O-Nonparticipants 

11 

39 

2,513.9 

112.0 





EQ2 Volunteers and Partnerships 

How did programs operate with regard to volunteers and partnerships? 

Volunteers 

Based on reports from program directors, volunteers were recruited from several 
sources as shown in Table 5. In sheer numbers, K-12 service learning programs were the larg- 
est source of volunteers (n = 1,148), followed by parents (n = 631) and higher education stu- 
dents service learning programs (n = 460; Figure 4). Using Table 5, here are the groups of 
volunteers with which program directors reported as the most successfully integrated into 
their programs. Higher education service-learning programs received an average rating of 3.8 
on a 4-point scale with 1 being no success and 4 being great success. Community organizations 
and K-12 service-learning programs also received high ratings at an average of 3.6. Finally, 
two smaller groups, AmeriCorps and faith-based organizations came in at 3.5— still quite a 
high rating on a 4-point scale. 
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Table 5. Number of Volunteers Recruits by Program and Success Rate 


Number 

programs 

with 

Percent of 

programs 

that 

recruited 

volunteers 

from this 


Percent of 

all 

Success 

volunteers 

source 

Number 

volunteers 

rate 


Overall 



3,117 

100.0 

3.5 

Service learning (K-12 students) 

18 

78.3 

1148 

36.8 

3.6 

Parents 

20 

87.0 

631 

20.2 

3.2 

Service learning (higher 
education students) 

14 

60.9 

460 

14.8 

3.8 

Community organizations 

17 

73.9 

230 

7.4 

3.6 

Faculty members 

16 

69.6 

184 

5.9 

3.4 

Local businesses 

18 

78.3 

171 

5.5 

3.4 

AmeriCorps 

11 

47.8 

104 

3.3 

3.5 

Faith-based organizations 

14 

60.9 

72 

2.3 

3.4 

Other 

5 

21.7 

55 

1.8 

3.8 

Local clubs (e.g. Kiwanis, Lions) 

10 

43.5 

35 

1.1 

3.2 

Senior corps 

6 

26.1 

27 

0.9 

3.5 

Data source: 2015 21st CCLC program director survey May-June 2015 



Figure 4. Percent of Volunteers by Source 

Data source: 2015 CCLC program director survey May-June 2015 
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Partnerships 

Based on reports from program directors, programs engaged in a variety of functions 
with partners, the three most frequently cited functions (see Table 6) were resources (212 or 
28.2%), programming (166 or 22.1%), and joint planning (135 or 18.0%). All of the partnership 
ratings were successful. 


Table 6. Number of Partners by Function and Success Rate 


Function 

Number of 
partnerships 

Percent of 
partnerships 

Mean 
success rating 

Training 

78 

10.4 

3.4 

Evaluation 

41 

5.5 

3.5 

Programming 

166 

22.1 

3.5 

Resources 

212 

28.2 

3.6 

Funding 

82 

10.9 

3.6 

Joint planning 

135 

18.0 

3.7 

Management 

38 

5.1 

3.8 


EQ3 Professional Development and Technical Assistance 

How well did professional development and technical assistance support 21st CCLC pro- 
grams, which formats are preferred, and what topics are most needed? 

Professional development 

As shown in Table 7, West Virginia program directors overall received the most train- 
ing from the WVDE, followed by counties/RESAs, and the U.S. Department of Education 
(USED). The best attended trainings were on, in descending order, family involvement, pro- 
gramming, and federal/ state requirement. 

Ratings of the effectiveness of the trainings were similar for all three sources (Table 7). 
The topics receiving the highest ratings included, in descending order, federal/state require- 
ments, programming, and collaboration. The lowest-rated topics included, in descending or- 
der, family involvement, and policy and advocacy (both with mean scores of 2.9), and program 
sustainability (2.6). 
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Table 7. Professional Development Training by Topic and Provider 



USED 

WVDE 

County/RESA 

Overall 

Professional 

development 

topic 

Number 

attended 

Average 

effective- 

ness 

rating 

Number 

attended 

Average 

effective- 

ness 

rating 

Number 

attended 

Average 

effective- 

ness 

rating 

Number 

attended 

Average 

effective- 

ness 

rating 

Programming 

8 

3.4 

20 

3.3 

17 

3.2 

45 

3.3 

Collaboration 

5 

3.4 

18 

3.1 

12 

3.3 

35 

3.3 

Communica- 

tions marketing 

5 

2.8 

10 

3.0 

12 

3.3 

27 

3.0 

Staff 

development 

6 

3.2 

16 

3.3 

14 

3.3 

36 

3.2 

Integrating 

afterschool 

with the 
regular school 
day 

5 

2.8 

17 

3.2 

11 

3.2 

33 

3.1 

Project 

management 

7 

3.0 

17 

3.1 

10 

3.2 

34 

3.1 

Federal/state 

requirements 

9 

3.6 

20 

3.4 

11 

3.5 

40 

3.5 

Family 

involvement 

10 

3.0 

21 

2.8 

15 

2.9 

46 

2.9 

Program 

sustainability 

5 

2.4 

22 

2.5 

12 

3.0 

39 

2.6 

STEM/STEAM 

6 

3.3 

21 

3.1 

12 

3.2 

39 

3.2 

Program 

evaluation 

7 

2.7 

17 

3.2 

10 

3.2 

34 

3.0 

Policy and 
advocacy 

7 

2.6 

18 

2.8 

10 

3.2 

35 

2.9 


Data source: 2015 21st CCLC program director survey May-June 2015 


Technical assistance 

Based on program director responses (see Table 8), site visits had the highest rating 
for helpfulness, with a 3.8 mean rating on a 4-point scale; emails and phone/conference calls, 
tied for second, both with mean ratings of 3.5. 


Table 8. Technical Assistance and Helpfulness Rating 


Type of TA 

N 

Helpfulness 

Site visit 

21 

3.8 

Email 

22 

3.5 

Phone/conference call 

22 

3.5 

CIPAS 

17 

3.3 

Action plan feedback 

14 

3.2 

Webinar 

17 

3.0 

Peer learning teams 

13 

2.5 


Data source: 2015 21st CCLC program director survey May-June 2015 
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Future needs for professional 
development and technical 
assistance 

Regarding professional 
development needed in the fu- 
ture, the most requested topics 
were program sustainability 
and staff development (Figure 
5). This is notable in light of the 
low ratings they gave for train- 
ing they had already received 
on this topic (see Table 7). 

Figure 6 shows a sum- 
mary of program director re- 
sponses to open-ended 
questions that asked about top- 
ics for which they need addi- 
tional professional develop- 
ment and technical assistance. 
Again, sustainability headed 
the list but other topics, not on 
the multiple-choice list, ranked 
very high, especially, profes- 
sional development and tech- 
nical assistance regarding 
personnel issues and training, 
and online data reporting and 
use. 




Program sustainability 

10 

3 





STEM/STEAM 

6 

2 

8 





Family involvement 

7 

3 

6 





Policy and advocacy 

7 

4 

4 





Program evaluation 

6 

5 

4 





Project management 

7 

3 

4 





Programming 

6 

3 

5 





Staff development 

9 

1 

3 





Federal/state requirements 

6 

2 4 





Integrating afterschool with... 

5 0 7 

Collaboration 

Communications/marketing 

5 

5 

1 5 

2 2 



0 5 10 15 20 

Number of times selected 

PD needed TA needed Info resources needed 


Figure 5. Most-Needed Topics for Profession Development, 
Technical Assistance, and Information Resources (Multiple 
Choice Items) 

Data source: 2015 CCLC program director survey May-June 2015 


Taking a deeper look at the top four expressed needs for profession development and 

technical assistance, these are comments from program directors, paraphrased for brevity. 

• Sustainability— (a) how to find fiscal support, (b) programs that include pay and non-pay 
participants, that is, sliding scales, (c) federal rules on reporting program income, (d) guid- 
ance on processes and good formats for presenting a sustainability plan, (e) how to create 
sustainability. 

• Personnel issues and training— (a) how to improve two-way communications with site 
staff, (b) staff engagement, (c) how to find quality staff for rural sites, especially coordina- 
tor positions, and (d) staff training, including finding great presenters, (e) basic, critical 
training resources for new staff, and (f) diversity training. 

• Online data and reporting— (a) online data collection, (b) data entry information, (c) use 
of data site, (d) a new user friendly PPICS (data system) and professional development on 
its usage, (e) working with the West Virginia Education Information System (WVEIS), and 
(f) training on WVEIS, particularly in how to access reports to use when seeking addi- 
tional funding opportunities. 
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Parent engagement— (a) 
working together on devel- 
oping materials that can be 
used at parent workshops 
and given to families to take 
home to work with their 
kids, and (b) finding ways of 
increasing parent engage- 
ment suitable for rural com- 
munities. 


0 
ao 

01 


Q. 

o 


Sustainability 
Personnel issues and training 
Online data reporting and use 
Parent engagement 
Programming for students 
Technology applications 
Communications and... 
School day integration with... 

Behavior management 
CIPAS/program evaluation 
Miscellaneous 



Figure 6. Most Needed Topics for Profession Development, 
Technical Assistance, and Information Resources (Open- 
Ended Questions) 

Data source: 2015 CCLC program director survey May-June 2015 


EQ4 Parent and Community Involvement 

What was the level of success in involving parents and community members? 

Figure 7 illustrates the 
level of success involving par- 
ents and community, based on 
responses from program direc- 
tors. More than 60% of pro- 
gram directors indicated they 
either had no family component 
(22%) or they had only slight 
success in their efforts to in- 
volve parents/guardians or 
other adult community mem- 
bers (39%). A small minority 
reports great success (4%). 

Data source: 2015 CCLC program director survey May-June 2015 



I No family 
component 

l Slight success 


: Moderate success 


Great success 


Figure 7. Level of Success Involving Parents and Community 
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EQ5 Improvement and Accountability Processes 


How helpful to 21st CCLC programs were improvement and 
accountability processes? 


Based on program di- 
rector responses (see Figure 8) 
59% felt monitoring visits were 
very helpful, 32% felt monitor- 
ing visits were moderately 
helpful, and 9% had no visits. 



Figure 8. Improvement and accountability process 

Data source: 2015 CCLC program director survey May-June 2015 


EQ6 Successes, Challenges, and Recommendations 

What do program directors view as their major successes, challenges, and recom- 
mendations for the future of the program? 

Program success 

Program directors completed open-ended questions which asked what two or three 
aspects of their programs had gone particularly well. Categories of responses emerged, as 
shown in Figure 9. We will take a closer look at specific comments in the top two categories— 
programs for students and better student attendance/participation, paraphrased as follows: 

• Programs for students— (a) Student engagement, such as making the content that we 
tutor hands on and fun for the students to learn the skills that they are missing during 
the regular school day, and getting students excited about learning and joining in 
unique experiences they may never have been exposed to; (b) wood working voca- 
tional enrichment; (c) STEM and STEAM programs, including for example, Ev3 Ro- 
botics, STEM/NASA SEMAA-training, Girl Talk with Dr. Yoost-Telemedicine, and on- 
going activities facilitated by grant partners and community organizations; (d) coop- 
eration with WVU Extension-Energy Express; (e) professional presenters for nutri- 
tion, character education, drug and alcohol prevention; (f) field trips to library, 
Humane Society, and state park; and (g) the richness of offerings at summer camp. 
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• Better student attendance/participation— (a) Students at one program really showed 
ownership over their site and the program, including being vocal in what programming 
worked for them and what did not; (b) the bond [students] formed was a family bond, 
and many improved attendance during the school day primarily so they could attend 
the afterschool program. 


Programs for students 
Better student attendance/participation 
Addition and quality of partnerships 

-t— 1 

cu Professional development of staff 

E Parent involvement 

o 

CJ 

Caring relationships with students 

> 

o Team building 

CUD 

CL) 

Supportive relationships with schools 
u 

Student achievement gains 
Staff tutoring/mentoring of students 
Other (1 mention each) 
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Number of mentions 


Figure 9. Summary of Program Successes 

Data source: 2015 CCLC program director survey May-June 2015 


Program challenges 

Turning now to open- 
ended questions about chal- 
lenges, at the top of the list 
were personnel issues, parent 
engagement and support, 
and the perennial funding 
and sustainability challenge 
(see Figure to). More specific 
detail about each of these 
three challenges is para- 
phrased as follows: 

• Personnel issues— (a) 
hiring, including find- 
ing summer staff 
members, finding 
quality new hire staff 
in this county, and 
hiring qualified staff 
at the new high school program; (b) training, including new site coordinators, project 


Personnel issues 
Parent engagement and support 
Funding and sustainability 
Student engagement and.. 
Inclement weather 
Misc. 

Linking/communicating with.. 

Transportation 
Managing centers 



2 4 6 8 

Number of mentions 


10 


Figure 10. Summary of Program Challenges 

Data source: 2015 CCLC program director survey May-June 2015 
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director, and new fiscal agent CEO/president; (c) laying off/ firing staff for various rea- 
sons throughout the year, and the impacts on students. 

• Improving parental engagement — some approaches included (a) putting a variety of 
new programs in place to make our school more inviting, (b) hosting more events out- 
side regular program hours when parents can be involved in the program, (c) forming 
a parent focus group, (d) sending text messages and video clips to parents, and (e) 
inviting more families to stay and help and communicate. 

• Funding and sustainability — challenges have included (a) overcoming the complexity 
of a freshman program and inexperience with grants; (b) delay in the grant awards; (c) 
cash flow, despite the efforts of WVDE staff to get drawdowns completed as soon as 
possible; and (d) increase in the number of families need assistance to attend the af- 
terschool program. 

Recommendations 

Lastly, program directors were asked to make recommendations for how to improve 
the program in the future (see Figure n). As you can see, improving the WVEIS reporting 
system was the most frequent recommendation. Most program directors who commented 
wrote about redundancies in the reporting, hard-to-use interfaces, and the inability to utilize 
data entered. One of the comments provided some detail about the nature of the problems 
with the WVEIS system: 

One of the biggest problems that we deal with is the WVEIS 21st CCLC data collection 
system. There needs to be a way to make the system more user friendly. There are too 
many menu and drop downs to try and figure out where the information needs to go. 

Also we have problems with being able to see the information once it is in the system. 

Right now it is set up like a black hole where the information goes in but you can’t get 
the information back out. 
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Figure 11. Recommendations 

Data source: 2015 CCLC program director survey May-June 2015 
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Another program director offered a suggestion about how to approach solving the 
problems; 

Convene a team of program staff members to continue the development of the WVEIS 
21 st CCLC Reporting System. This group would make recommendations for continued 
system improvements— such as system navigation, and report generation. 

Discussion and Conclusions 

The 21st Century Community Learning Centers (21 st CCLC) program in West Virginia 
continues to touch many young lives in the state, at its best providing a safe haven where stu- 
dents learn both experientially and academically. Teachers report student improvement in 
turning in high quality homework assignments on time, behavior during class and with other 
students, academic performance, and motivation to learn. With only one year of assessment 
data due to the change to the new West Virginia General Summative Assessment, it is not 
possible to confirm improvements in academic performance using quasi-experimental meth- 
ods, but that may be possible in 2015-2016, once there are two years of testing data. 

Programs had well-identified sources of volunteers and partners, and have access to 
ongoing professional development and technical assistance. Program directors appreciate the 
feedback they receive during monitoring visits, as they work to continuously improve their 
programs. Parent and community involvement is reported to be an ongoing challenge, alt- 
hough a small number of programs report some success in involving parents using various 
outreach, programming, and technological methods. 

Programs struggle with reductions in funding and achieving sustainability. However, 
the most persistent and vociferous complaints raised by program directors relate to the inad- 
equate data systems that continue to plague the 21st CCLCs and their evaluation. These inad- 
equacies have resulted in the following negative impacts: 

• Poor database interfaces cost staff tremendous amounts of wasted time in meeting 
their reporting obligations— time that could be far better spent on improving programs 
for children and families; and 

• Staff cannot access data they have inputted, thus impeding their ability to use data to 
plan for their programs. 

West Virginia Department of Education (WVDE) staff performing this evaluation also experi- 
enced a lack of access to data needed for conducting teacher surveys, resulting in delays that 
reduced response rates and produced lower-quality data about impacts on students. 

The WVDE can enhance the accountability and evaluation of the 21 st CCLC program 
by addressing the data system challenges reported by the program directors. A strong evalua- 
tion plan based on a quality data system is essential to the success of the 21 st CCLC program. 
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Methods Used to Address EQ1 

Descriptive statistics using West Virginia Department of Education (WVDE) 21 st Century 
Community Learning Center (21 st CCLC) database 

The following statistics were derived using data from the WVDE 21 st CCLC database: 

• The number of students and their distribution by grade level 

• The average student attendance (dose strength) by program 

Data collected in this database were submitted by the site managers and program directors on 
a daily basis. 

Descriptive statistics using online 21 st CCLC Teacher Survey 

Based on responses to the Teacher Survey (see Appendix B), we calculated an effect 
size of the program on various behaviors by using a retrospective pre-post measure. Based on 
responses to the Teacher Survey and the WVDE 21 st CCLC database, we determined if there 
was a relationship between reasons for referral and the length of time students spent in the 
program beyond 30 days. 

Population characteristics and sampling procedures 

The sample for the teacher survey started with the teachers of those students who par- 
ticipated in the 21 st CCLC program for at least 30-days whose parents had given passive con- 
sent. Consent forms were given to parents at the time they registered their children; parents 
were instructed that if they agreed to have their children be part of the evaluation, no action 
was necessary. If they denied consent, they returned the signed form, which will be kept in 
program directors' offices until April of each year, and then sent to the WVDE Office of Re- 
search, Accountability, and Data Governance. Denial of consent was logged into the 21st CCLC 
database maintained at WVDE, using a checkbox added to the system for the 2015-2016 
school year and thereafter. 

Quasi-experimental study 

We conducted a quasi-experimental examination of existing student assessment data 
obtained from the West Virginia Education Information System (WVEIS) in mathematics and 
English/language arts (ELA) for students who participated in a 21st CCLC during the one- 
school-year study period. 

Population characteristics 

The treatment group consisted of students who participated in at least 30 days of 21st 
CCLC intervention as documented in the WVDE 21 st CCLC database. The comparison group 
was matched using propensity scores, matching a variety of demographic and performance 
covariates. Analyses were conducted to examine both within- and between-group differences 
in student achievement. 
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Sampling procedures 

From attendance records submitted by site managers to the WVDE 21 st CCLC database 
(not the teacher survey dataset), we identified students who 

• were in Grade 4 or above 

• who had test records for both school years 

• who received scale scores for both mathematics and ELA 

• who had a complete set of demographic covariate variables to be used during 
matching 

• who were not retained from one year to the next. 

After removing students who do not meet all of these criteria, we were left with our final sam- 
ple. 

We used propensity score matching (PSM) to select a matched comparison group for 
each grade level. This methodology used logistic regression to select a comparison group that 
closely matched the treatment group on a variety of observed covariates. First, a binary indi- 
cator showed whether or not each student in the state participated in 21st CCLC during the 
study school year. Group 1 was defined as the treatment group (those students who attended 
30 or more days in a 21st CCLC) and Group o was the control group (those students who did 
not participate in 21st CCLC during the school year studied). We will then specify the PSM 
models, which derived conditional probabilities for each student by regressing the binary 
group membership variable on the following covariates: (a) prior academic achievement in 
both mathematics and ELA, (b) gender, (c) race/ethnicity, (d) free/reduced-price lunch eligi- 
bility, and (e) special education eligibility. Grade level was held constant by conducting match- 
ing independently within each grade. Thus, in this study the propensity score represents the 
predicted probability that a given student would attend 30 days of 21st CCLC based on this set 
of pre-intervention covariates. Finally, we used nearest-neighbor matching to select the most 
appropriate match for each 21st CCLC student. Verification analyses was conducted to check 
that this matching methodology identified an adequately balanced comparison group for hy- 
pothesis testing 1 . 

Measures and covariates 

This portion of the study includes an examination of student achievement data from 
the West Virginia General Summative Assessment (WVGSA). We analyzed individual stu- 
dents’ scale scores, and gain scores in both mathematics and ELA. Gain scores were opera- 
tionalized as the change in student scale scores from the previous school year to the study 
school year, with the expectation that students who participated in 21st CCLC during the study 


1 We will use chi squared analyses to verify the two groups did not differ on categorical demo- 
graphic variables. We will use independent samples t-tests to verify the two groups did not differ on 
prior academic achievement in ELA and mathematics. 
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school year would experience differential gains when compared with similar students not 
known to have participated in 21st CCLC. 2 

Covariates in this study included students’ gender, race/ethnicity, special education 
eligibility, free/reduced-price lunch eligibility, and prior academic achievement in mathemat- 
ics and ELA. 

Data collection methods 

All data for the quasi-experimental portion of this study was collected from two 
sources— the 21st CCLC database and WVEIS general summative assessment records file, both 
maintained by the WVDE. 

Research design 

Ten sets of analyses were ran using t tests or ANOVAs, to answer the following four 
questions: 

1. What was the impact of 21st CCLC participation on l-year mathematics and ELA 
gains? (Sets 1 and 2) 

2. What was the impact of 21st CCLC participation on end-of-year mathematics and ELA 
achievement? (Sets 3 and 4) 

3. What were the year-to-year changes in mathematics and ELA achievement for each 
group (CCLC participants and nonparticipants) independently? 2 (Sets 5-8) 

4. Were mathematics and ELA scores attained by 21st CCLC participants significantly 
different from those attained by nonparticipants? (Sets 9 and 10) 

Details about these tests are described next. 

Question A (Sets 1 and 2) were addressed using a series of independent samples t tests. 
These analyses used group membership as the independent variable and mean WVGSA scale 
score gains from the one school year to the next (e.g., 2015-2016 to 2016-2017) in mathemat- 
ics and ELA as the outcome variables. Each grade level and content area combination were 
tested independently to estimate impact of the 21st CCLC program. In sum, we conducted 16 
tests: 

• Eight tests (one per grade for Grades 4-11) examined the impact of group member- 
ship on year-to-year WVGSA mathematics gains to determine whether 21st CCLC 
students (treatment) experienced greater gains than students who did not receive 
the treatment (control). 

• Eight tests (one per grade for Grades 4-11) examined the impact of group member- 
ship on year-to-year WVGSA ELA gains to determine whether 21st CCLC students 
(treatment) experienced greater gains than students who did not receive the treat- 
ment (control). 


2 We were unable to perform this analysis for the 2014-2015, the first year the new WVGSA 
based on Smarter Balanced was used in West Virginia. Beginning in 2015-2016, this analysis will again 
be possible, as we will have two years of data. We were, however, able to make a comparison between 
the participating and nonparticipating students’ growth data. 
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Question B was addressed using a series of independent samples t tests (Sets 3 and 4). 
These analyses used group membership as the independent variable and mean study year 
WVGSA outcomes in mathematics and ELA as the outcome variables. Each grade level and 
content area combination were tested independently to estimate impact of the 21st CCLC 
program in the study year. In sum, we conducted 16 tests: 

5. Eight tests (one per grade for Grades 4-11) examined the impact of group membership 
on study year WVGSA mathematics outcomes to determine whether 21st CCLC stu- 
dents (treatment) scored higher than students who did not receive the treatment (con- 
trol). 

6. Eight tests (one per grade for Grades 4-11) examined the impact of group membership 
on study year WVGSA ELA outcomes to determine whether 21st CCLC students (treat- 
ment) scored higher than students who did not receive the treatment (control). 

Question C was addressed using a series of paired t tests (Sets 5 -8). Analysis Sets 5 
and 6 focused on the 21 st CCLC students and used time as the independent variable and stu- 
dents’ mean study year and prior year WVGSA scale scores in mathematics and ELA as the 
outcome variables. Each grade level and content area combination were tested independently. 
In sum, we conducted 16 tests: 

7. Eight tests (one per grade for Grades 4-11) examined the impact of time on WVGSA 
outcomes to determine whether 21st CCLC students exhibited higher achievement in 
mathematics during the study school year when compared to their own mathematics 
results for the prior academic year. 

8. Eight tests (one per grade for Grades 4-11) examined the impact of time on WVGSA 
outcomes to determine whether 21st CCLC students exhibited higher achievement in 
ELA during the study school year when compared to their own ELA results for the 
prior academic year. 

Analysis Sets 7 and 8 also used a series of paired t tests but focused on the control 
group (non 21 st CCLC students). These analyses used time as the independent variable and 
mean study year and prior year WVGSA scale scores in mathematics and ELA as the outcome 
variables. Each grade level and content area combination were tested independently. In sum, 
we conducted 16 tests: 

9. Eight tests (one per grade for Grades 4-11) examined the impact of time on WVGSA 
outcomes to determine whether control group students exhibited higher achievement 
in mathematics during the study school year when compared to their own 
mathematics results for the prior academic year. 

10. Eight tests (one per grade for Grades 4-11) examined the impact of time on WVGSA 
outcomes to determine whether control group students exhibited higher achievement 
in ELA during the study school year when compared to their own ELA results for the 
prior academic year. 

Question D was addressed in analysis Sets 9 and 10 using repeated measures analysis 
of variance (RM ANOVA) tests. These analyses used two predictor variables, group member- 
ship and time, as independent variables predicting the outcome of WVGSA performance in 
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mathematics and ELA. In these analyses we looked for a significant interaction effect to indi- 
cate one group scored differently from the other over time. In sum we conducted 16 tests: 

11. Eight tests (one per grade for Grades 4-11) examined the interaction of group and time 
on WVGSA mathematics outcomes to determine whether students in the treatment 
group scored significantly higher than students in the control group over time. 

12. Eight tests (one per grade for Grades 4-11) examined the interaction of group and time 
on WVGSA ELA outcomes to determine whether students in the treatment group 
scored significantly higher than students in the control group over time. 

Methods Used to Address EQ2.-EQ6. 

The final evaluation questions were addressed using descriptive statistics and qualita- 
tive analysis of responses to the Online 21st CCLC Program Directors Survey (see Appendix 
B). 
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See next pages. 
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VOLUNTEER PROGRAMS 


If your program uses volunteers, which sources have you used, how many, and with what level of success? 


Not 

Number of applicable Slight Moderate Great 

volunteers (none used) No success success success success 

AmeriCorps (AmeriCorps Promise Fellow, AmeriCorps VISTA, 
Citizen Community Corps) 


o o o o o 

Senior Corps (retired and senior volunteers, Foster Grand- 
parent Program) 


o o o o o 

Service learning (K-12 students) 


o o o o o 

Service learning (higher education students) 


o o o o o 

Local businesses 


o o o o o 

Community organizations 


o o o o o 

Faith-based organizations 


o o o o o 

Parents 


o o o o o 

Faculty members 


o o o o o 

Local clubs (e.g. Kiwanis, Lions) 


o o o o o 

Other 


o o o o o 
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PARTNERSHIPS 


If your program uses partners, please indicate for each of the following forms of support, how many partners you have engaged and 
at what level of success. 



Number of 
partners 

Not applicable 
(none used) 

No success 

Slight success 

Moderate success 

Great success 

Joint planning 


o 

o 

o 

o 

o 

Resources 


o 

o 

o 

o 

o 

Funding 


o 

o 

o 

o 

o 

Programming 


o 

o 

o 

o 

o 

Training 


o 

o 

o 

o 

o 

Management 


o 

o 

o 

o 

o 

Evaluation 


o 

o 

o 

o 

o 

Other 


o 

o 

o 

o 

o 
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PROFESSIONAL DEVELOPMENT-COUNTY OR RESA 


For each of the following topics, please rate the level of effectiveness of any COUNTY OR RESA training you have 
received during this program year. 

Please check the preferred format(s) if you 
need more information on this topic from 

the COUNTY or RESA. 


Not applica- 
ble (none at- 
tended) 

Not effective 

Slightly 

effective 

Moderately 

effective 

Highly 

effective 

Professional 

development 

Technical 

assistance 

Information 

resources 

Programming 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Collaboration 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Communications/marketing 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Staff development 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Integrating afterschool with the regular 
school day 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Project management 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Federal/state requirements 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Family involvement 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Program sustainability 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Stem/steam 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Program evaluation 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Policy and advocacy 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Other 

o 

o 

o 

o 

o 

□ 

□ 

□ 
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PROFESSIONAL DEVELOPMENT-WEST VIRGINIA DEPARTMENT OF EDUCATION 


For each of the following topics, please rate the level of effectiveness of any West Virginia Department of 
Education training you have received during this program year. 

Please check the preferred format(s) if you 
need more information on this topic from 

the WVDE. 


Not applica- 
ble (none at- 
tended) 

Not effective 

Slightly 

effective 

Moderately 

effective 

Highly 

effective 

Professional 

development 

Technical 

assistance 

Information 

resources 

Programming 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Collaboration 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Communications/marketing 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Staff development 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Integrating afterschool with the regular 
school day 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Project management 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Federal/state requirements 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Family involvement 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Program sustainability 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Stem/steam 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Program evaluation 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Policy and advocacy 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Other 

o 

o 

o 

o 

o 

□ 

□ 

□ 
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PROFESSIONAL DEVELOPMENT-UNITED STATES DEPARTMENT OF EDUCATION 


For each of the following topics, please rate the level of effectiveness of any United States Department of Educa- 
tion training you have received during this program year. 

Please check the preferred format(s) if you 
need more information on this topic from 

the U.S. Department of Education. 


Not applica- 
ble (none at- 
tended) Not effective 

Slightly 

effective 

Moderately 

effective 

Highly 

effective 

Professional 

development 

Technical 

assistance 

Information 

resources 

Programming 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Collaboration 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Communications/marketing 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Staff development 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Integrating afterschool with the regular 
school day 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Project management 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Federal/state requirements 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Family involvement 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Program sustainability 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Stem/steam 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Program evaluation 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Policy and advocacy 

o 

o 

o 

o 

o 

□ 

□ 

□ 

Other 

o 

o 

o 

o 

o 

□ 

□ 

□ 
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TECHNICAL ASSISTANCE 


Please indicate level of helpfulness of each of the following forms of technical assistance you may have received. 



Not applicable 
(none received) 

Not 

helpful 

Slightly 

helpful 

Moderately helpful 

Very 

helpful 

Email 

o 

o 

o 

o 

o 

Phone call/conference call 

o 

o 

o 

o 

o 

Webinar 

o 

o 

o 

o 

o 

Action plan feedback 

o 

o 

o 

o 

o 

Peer learning teams 

o 

o 

o 

o 

o 

Site visit 

o 

o 

o 

o 

o 

Cl PAS 

o 

o 

o 

o 

o 
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PARENT/COMMUNITY INVOLVEMENT 


Parent/Community Involvement 



Not applicable 
(no family compo- 

No 

Slight 


Great 


nents) 

success 

success 

Moderate success 

success 

Please rate your level of success in involving 
parents/guardians or other adult community 
members. 

o 

o 

o 

o 

o 

How many adults have participated in your programs by . . . 


Attending programs designed for them? 



Helping with program planning? 



Participating in program evaluations? 



Helping deliver services? 
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CONTINUOUS IMPROVEMENT PROCESS 


How helpful has the Continuous Improvement Process for After School (CIPAS) been to your program? 


Not applicable 

Not very helpful 

Neutral 

Moderately helpful 

Very helpful 



It validated what 1 was doing 


1 learned about my program 


1 did not learn from the pro- 

right, but 1 could use more in- 

1 learned my program's 

and received useful recom- 

1 did not complete process. 

cess. 

formation to improve. 

strengths and challenges 

mendations. 

o 

o 

o 

o 

o 


PROGRAM MONITORING PROGRESS 


How helpful have the monitoring visits by WVDE staff to your site been this year? 


Not applicable 

Not very helpful 

Neutral 

Moderately helpful 

Very 

helpful 

My program was not visited 
by a WVDE monitor this year. 

Information presented was 
incomplete or inaccurate and 

1 learned nothing new. 

The information presented 
was accurate, but 1 learned 
nothing new. 

1 learned areas of strength 
and challenge and identified 
ways to overcome barriers. 

1 learned about my program 
and centers, shared the re- 
port with staff and stakehold- 
ers, and identified new 

resources. 

o 

o 

o 

o 

o 
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Program Successes/Program Challenges/Recommendations 


In the past year, what two or 
three aspects of your program 
have been going particularly well? 

1 . 

2. 

3. 

In the past year, what have been 
the two or three biggest chal- 
lenges facing your program? 

1 . 

2. 

3. 

What three specific professional 
development or technical assis- 
tance topics would you find most 
helpful for WVDE staff to deliver 
during the upcoming school year? 

1 . 

2. 

3. 

What recommendations do you 
have for improving the statewide 
21st CCLC program overall? 

1 . 

2. 

3. 


21st Century Community Learning Centers, 2014-2015 / 33 


Appendix B. Survey Questionnaires 


Teacher Survey 


TEACHER SURVEY: 21 st CENTURY COMMUNITY LEARNING CENTER 


Teacher email 



Student ID 


Grant ID 


Center ID 



Earlier this year 
this student's perfor- 
mance was: 


If not acceptable, what level of change has this student shown 
during the course of the year? 


Areas of performance: 

Acceptable 

Not 

acceptable 

Significant 

improve- 

ment 

Moderate 

improve- 

ment 

Slight im- 
prove-ment 

No 

change 

Slight de- 
cline 

Moderate 

decline 

Significant 

decline 

1. Turning in his/her homework on 
time 

o 

o 

o 

o 

o 

o 

o 

o 

o 

2. Completing homework to your satis- 
faction 

o 

o 

o 

o 

o 

o 

o 

o 

o 

3. Participating in class 

o 

o 

o 

o 

o 

o 

o 

o 

o 

4. Volunteering (for example, for extra 
credit or more responsibilities) 

o 

o 

o 

o 

o 

o 

o 

o 

o 

5. Attending class regularly 

o 

o 

o 

o 

o 

o 

o 

o 

o 

6. Being attentive in class 

o 

o 

o 

o 

o 

o 

o 

o 

o 

7. Behaving well in class 

o 

o 

o 

o 

o 

o 

o 

o 

o 

8. Academic performance 

o 

o 

o 

o 

o 

o 

o 

o 

o 

9. Coming to school motivated to learn 

o 

o 

o 

o 

o 

o 

o 

o 

o 

10. Getting along well with other stu- 
dents 

o 

o 

o 

o 

o 

o 

o 

o 

o 
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Informed Consent of Parents/Guardians 

[Printed on WVDE letterhead] 

2015-2016 Evaluation of West Virginia’s 
21 st Century Community Learning Center Program 

Parent/Guardian Informed Consent 

I understand that the afterschool program my child will attend will be evaluated by the 
West Virginia Department of Education (WVDE], The purpose of the evaluation study is to find 
out how well the program is working. What the WVDE learns from this study may help improve 
the program in the future. Later this school year, we would like to ask your child's teacher about 
the amount of progress your child has made. Any information we would gather would be pro- 
tected and your child would never be identified. The information provided would be combined 
with information from others, and reported as a group. 

Allowing your child to take part in this study in the way just described will put your child 
at no more risk than he or she would experience during any normal day. Although your child may 
not benefit directly by being part of the study, it is possible that because of what we learn, the 
program may improve to better meet his or her needs or the needs of other students. 

Neither you nor your child will receive any money or other reward for taking part in this 
study. Allowing your child to be part of the study is completely voluntary. If you decide not to 
allow your child to be part of it, there will be no penalties or loss of benefits to you or your child. 

To allow us to collect this information from your child’s teacher there is no action 
you need to take. Thank you! 

If you do NOT want your child to be part of the study, just fill in the information 
below and return this form to the afterschool program coordinator. 

Do NOT include my child in the evaluation study. 

Child's name (please print]: 

Parent/guardian signature: Date: 

Name of afterschool program: (to be filled in by program staff): 


For more information about the education program we are studying, you may contact Josh Asbury (304- 
872-6440, imasburv(5)kl2.wv.us'l or Benitez Jackson (304-256-4712; bljackso@kl2.wv.us). If you have 
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questions about this evaluation study, you may contact Patricia Hammer (304-558-2546; pharn- 
mer(a>kl2.wv.us ~l. This study has been reviewed and approved by the West Virginia Department of Educa- 
tion Institutional Review Board (1RB-WVDE-XXX)). If you want to know more about the review of this 
study, you may contact the WVDE IRB cochair, Andy Whisman f awhisman.kl2.wv.us ~l. 

Informed Consent for Teacher Survey 


By filling out this survey, you are agreeing to take part in an evaluation study. The 
purpose of the study is to evaluate the effectiveness of the 21 st Century Community Learning 
Center (21 st CCLC) program. What we learn from this study may help improve this program 
or other afterschool programs in the future. Your participation in the study is limited to com- 
pleting a brief survey about each of your students, which should not take more than 3-4 
minutes for each student. You will be presented with a series of items and asked to indicate 
your responses by either checking off a rating. 

Taking part in this study will put you at no more risk than you would experience during 
any normal day. Although you may not benefit directly by taking part in the study, it is possible 
that because of what we learn, the program may improve to better meet your needs or the 
needs of students. Your responses to this survey will be protected and will never be revealed 
as coming from you. All responses will be combined and reported as a group. 

You will receive no monetary or other reward for taking part in this research study. 
Filling out the survey is completely voluntary. If you decide not to take part or to stop at any 
time, there will be no penalties or loss of benefits to you. For more information about the 
education program we are studying, you may contact Josh Asbury (304-872-6440, imas- 
burv@k12.wv.usf or Benitez Jackson (304-256-4712; bljackso@k12.wv.us). If you have ques- 
tions about this evaluation study, you may contact Patricia Hammer (304-558-2546; 
phammer@ki2.wv.us) . This study has been reviewed and approved by the West Virginia De- 
partment of Education Institutional Review Board (IRB-WVDE-XXX)). If you want to know 
more about the review of this study, you may contact the WVDE IRB cochair, Andy Whisman 
(awhisman.ki2.wv.us) . 

Thank you for taking part in this important effort. 
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Informed Consent for Program Director Survey 


By filling out this survey, you are agreeing to take part in an evaluation study. The 
purpose of the study is to find out how well various aspects of the 21 st Century Community 
Learning Center program are working and to gather information that can be used to guide the 
program in the future. What we learn from this study may help improve the program or other 
education programs. To be part of the study, all you need to do is complete a survey by check- 
ing off your answers to the questions. Some questions ask you to write an answer in a text box. 
Filling out the survey should not take more than 30 minutes. 

Taking part in this study will put you at no more risk than you would experience during 
any normal day. Although you may not benefit directly by taking part in the study, it is possible 
that because of what we learn, the program may improve to better meet your needs or the 
needs of students. Your responses to this survey will be protected and will never be revealed 
as coming from you. Your responses will be combined with responses from others, and re- 
ported as a group. 

You will receive no money or other reward for taking part in this research study. Filling 
out the survey is completely voluntary. If you decide not to take part or to stop at any time, 
there will be no penalties or loss of benefits to you. For more information about the education 
program we are studying, you may contact Josh Asbury (304-872-6440, jmasbury(5)kl2.wv.us) or Be- 
nitez Jackson (304-256-4712; bljackso@kl2.wv.us). If you have questions about this evaluation study, 
you may contact Patricia Hammer (304-558-2546; phammer(5)kl2.wv.us) . This study has been re- 
viewed and approved by the West Virginia Department of Education Institutional Review Board (IRB- 
WVDE-XXX)). If you want to know more about the review of this study, you may contact the WVDE 
IRB cochair, Andy Whisman ( awhisman.l<12.wv.us ). 

Thank you for taking part in this important effort. 
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